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Abstract 


Department  of  Defense  (DoD)  spending  has  been  steadily  increasing  ever 
since  the  early  1990s.  During  that  period,  the  acquisition  workforce  has  steadily 
declined.  This  situation  resulted  in  an  undermanned  and  undertrained  contracting 
workforce  with  an  increased  workload.  With  the  workforce  spread  thin,  lapses  in 
contracting  processes  occurred.  As  a  result  of  these  issues,  in  2008,  the  DoD 
established  the  requirement  for  independent  management  reviews,  or  peer  reviews, 
of  contractual  actions. 

Since  the  onset  of  the  peer-review  requirement,  the  Defense  Procurement 
Acquisition  Policy  (DPAP)  has  maintained  a  database  of  peer-review  results.  Data 
analytics  were  used  to  analyze  the  frequency  of  occurrences  of  the  data  elements 
within  the  DPAP  database  of  peer-review  results  in  an  effort  to  answer  two  research 
questions.  First,  are  there  trends  within  the  peer-review  results  of  DoD-level  peer- 
reviewed  contracts?  Second,  are  any  trends  identified  related  to  the  competency 
gaps  identified  in  the  2007  Department  of  Defense  Contracting  Workforce 
Competency  Assessment  Final  Report? 

Trends  within  the  data  elements  present  in  the  DPAP  database  of  peer- 
review  results  were  identified.  Certain  categories  garnered  more  attention  of  the 
peer-review  teams.  Based  on  trends  identified,  recommendations  are  provided  to 
improve  the  overall  usefulness  of  the  DPAP  database  of  peer-review  results. 

Keywords:  Peer  Reviews,  DPAP,  Contracts,  Contractual  Actions, 
Contracting  Workforce,  Contracting  Core  Competencies 
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I.  INTRODUCTION 


A.  BACKGROUND 

Department  of  Defense  (DoD)  spending  has  been  steadily  increasing  ever 
since  the  early  1990s.  During  the  same  time  period,  the  acquisition  workforce  has 
steadily  declined.  The  2007  Gansler  Commission  report  on  Army  contracting  noted 
that  over  a  12-year  period,  contracting  actions  had  risen  350%  while  the  Army 
contracting  workforce  had  been  reduced  by  50%  (Gansler,  2007).  This  situation 
resulted  in  an  undermanned  and  undertrained  contracting  workforce,  with  a  vastly 
increased  workload.  Within  the  DoD,  workload  per  GS-1 102  (Contract  Specialist) 
staff  member  rose  from  an  annual  average  of  $6.4  million  in  contract  actions  in  fiscal 
year  (FY)  1996  to  nearly  $13  million  in  FY  2005  (Girovasi,  2007).^  With  the  onset  of 
overseas  contingency  operations  in  Iraq  and  Afghanistan,  demand  for  the  acquisition 
workforce  spiked.  Figure  1  highlights  the  increase  in  spending  along  with  the 
corresponding  decrease  in  the  acquisition  workforce. 
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Figure  1.  DoD  Acquisition  Trends 

(Gansler,  2007) 


^  As  the  dollar  value  of  a  contract  action  increases,  so  does  the  workload  required  to  process  it.  For 
example,  Micropurchase  procedures  apply  under  $3,000,  and  Simplified  Acquisition  Procedures 
apply  from  $3,000  up  to  $150,000  (FAR  13.003(b)(1),  2014).  Additional  workload  is  generated  when  a 
contractor  is  required  to  submit  Certified  Cost  or  Pricing  Data  for  purchases  above  $700,000  when  no 
exceptions  apply  (FAR  15.403-4(a)(1),  2014). 
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With  the  workforce  spread  thin,  lapses  in  contracting  processes  began  to 
occur.  A  Government  Accountability  Office  (GAO)  report,  GAO-05-274,  Contract 
Management:  Opportunities  to  improve  Surveillance  on  DoD  Service  Contracts, 
issued  in  March  2005,  reviewed  90  service  contracts.  Out  of  these  90  contracts,  the 
GAO  (2005)  found  insufficient  surveillance  on  24.  From  those  24  contracts,  15  had 
no  surveillance  whatsoever  (GAO,  2005).  The  Gansler  Commission  report  revealed 
that,  as  of  2007,  83  Army  criminal  investigations  relating  to  contract  fraud  were 
ongoing  in  Iraq,  Afghanistan,  and  Kuwait.  At  the  same  time,  23  government 
employees  (both  civilian  and  military)  were  charged  or  indicted  in  federal  court.  The 
contracts  affected  represented  over  $6  billion  in  value  (Gansler,  2007). 

As  a  result  of  these  and  other  issues  discussed  during  the  literature  review 
chapter,  the  DoD  established  the  requirement  for  independent  management  reviews, 
or  peer  reviews,  of  contractual  actions  in  2008.  The  purpose  of  the  peer-review 
requirement  was  to  help  address  the  shortfalls  in  manning  and  experience  within  the 
DoD  contracting  community  by  ensuring  policy  and  regulations  were  followed  in  a 
consistent  manner,  and  that  the  available  experience  was  shared  across  the 
contracting  workforce  (Assad,  2008).  Identifying  and  tracking  the  trends  within  the 
data  provided  by  the  peer-review  process  is  critical  to  identifying  systemic  problems 
within  both  the  training  of  the  DoD’s  contracting  workforce  and  how  the  contracting 
workforce  is  operating. 

B.  PURPOSE 

The  primary  purpose  of  this  project  was  to  examine  the  Defense  Procurement 
Acquisition  Policy  (DPAP)  database  of  peer-review  results  and  identify  any  trends 
within  the  data.  This  project  was  not  intended  to  be  an  exhaustive  study  of  the  DPAP 
database  of  peer-review  results.  Rather,  it  was  contemplated  as  a  way  to  identify 
overarching  trends  for  further  analysis  in  the  future.  There  is  one  secondary  goal 
associated  with  this  project:  examine  any  relationships  between  trends  observed 
within  the  DPAP  database  of  peer-review  results  and  the  competency  gaps  by  the 
Contracting  Competency  Model  (CCM)  in  the  Department  of  Defense  Contracting 
Workforce  Competency  Assessment  Final  Report  (DPAP,  2007). 

C.  RESEARCH  QUESTIONS 

In  an  effort  to  investigate  a  previously  unexplored  topic,  I  crafted  two  research 
questions: 

•  Are  there  trends  within  the  peer-review  results  of  DoD-level  peer- 
reviewed  contracts? 
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•  Are  any  trends  identified  related  to  the  competency  gaps  identified  in 
the  Department  of  Defense  Contracting  Workforce  Competency 
Assessment  Final  Report  (DPAP,  2007)? 

D.  BENEFITS  AND  LIMITATIONS  OF  THIS  PROJECT 

The  results  of  this  project  can  be  used  to  target  and  strengthen  areas  of 
training  that  are  leading  to  systemic  issues  within  the  contracting  process.  It  can  also 
be  used  to  identify  areas  of  the  contracting  process  that  are  simply  being  neglected 
and  would  benefit  from  additional  emphasis  by  contracting  officers  and  their 
supervisors.  Improved  understanding  of  the  trends  within  the  DPAP  database  of 
peer-review  results  will  enable  policy-makers  to  structure  decisions  appropriately  to 
increase  both  the  efficiency  and  effectiveness  of  the  DoD  contracting  workforce. 

The  limitation  of  this  project  rests  with  the  fact  there  were  only  288  entries 
within  the  DPAP  database  of  peer-review  results.  Thus,  it  was  not  possible  to  obtain 
a  larger  population  or  sample  size  to  analyze. 

E.  RESEARCH  METHODOLOGY 

The  research  began  with  a  literature  review.  The  literature  review  addresses 
why  DoD  conducts  peer  reviews  and  addresses  corporate  procurement  review 
practices,  development  of  the  peer-review  requirement,  and  guidance  on  conducting 
peer  reviews.  I  approached  the  research  questions  by  analyzing  the  DPAP  database 
of  peer-review  results  in  terms  of  frequency  of  occurrences  of  various  data  aspects, 
including  Category,  Type  of  Contract,  Review  Phase,  and  Type  of  Feedback.  I 
examined  the  narrative  comments  section  from  a  qualitative  standpoint  to  identify 
any  recurring  themes  within  the  comments. 

F.  ORGANIZATION  OF  REPORT 

The  report  consists  of  five  chapters.  Chapter  I  contains  a  brief  background, 
the  purpose  of  the  research,  research  questions  examined,  relative  benefits  and 
limitations  of  the  project,  and  an  overview  of  the  research  methodology.  Chapter  II  is 
the  literature  review,  which  is  primarily  concerned  with  addressing  the  questions  of 
why  we  peer  review  contractual  actions,  how  the  requirement  was  developed,  and 
how  it  has  been  implemented.  To  that  end,  the  history  of  the  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA)  is  examined.  I  review  multiple  GAO  and 
commission  reports  that  reveal  the  decline  in  the  experience  and  manning  of  the 
acquisition  workforce,  coupled  with  the  rise  in  spending  activities  and  the  resultant 
issues  that  caused.  Chapter  III  discusses  where  the  data  originated  and  how  it  was 
analyzed.  Chapter  IV  provides  the  results  from  the  analysis  of  the  data.  Chapter  V 
contains  the  summary  of  the  project,  recommendations  for  improving  the  DPAP 
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database  of  peer-review  results,  and  areas  for  further  research  relating  to  the 
findings  of  this  project. 

G.  SUMMARY 

Chapter  I  provided  the  background  information  on  the  peer-review 
requirement  and  the  purpose  of  the  project,  stated  the  research  questions  and  the 
benefits  and  limitations  of  the  project,  and  briefly  addressed  the  methodology  and 
organization  of  the  written  report.  The  next  chapter  contains  the  literature  review. 
The  literature  review  addresses  why  the  DoD  conducts  peer  reviews  of  contractual 
actions.  It  also  addresses  corporate  procurement  review  practices,  development  of 
the  peer-review  requirement,  and  guidance  on  conducting  peer  reviews. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  previous  chapter  discussed  the  basis  of  this  research.  In  this  chapter,  I 
examine  why  the  DoD  conducts  peer  reviews  of  contractual  actions.  I  also  examine 
development  of  the  peer-review  requirement,  and  guidance  on  conducting  peer 
reviews.  Elements  of  this  chapter  include  explanations  of  the  DoD’s  initiation  of 
required  peer  reviews,  the  definition  of  a  peer  review,  and  the  execution  of  peer 
reviews.  Additionally,  I  review  how  both  public  and  private  procurement  agencies 
evaluate  performance. 

B.  WHY  PEER  REVIEW? 

1.  Introduction 

To  understand  the  reasoning  behind  the  peer-review  requirement,  it  is 
necessary  to  look  back  to  the  beginnings  of  the  modern  defense  acquisition 
workforce,  which  can  be  traced  back  to  the  early  1990s,  with  the  passage  of  the 
Defense  Acquisition  Workforce  Improvement  Act  (DAWIA,  1990).  The  history  since 
1990  develops  a  story  of  the  development  of  a  truly  professional  workforce  and  how 
it  ran  into  issues  of  cutbacks  in  an  era  of  persistent  defense  spending  cuts.  This  led 
to  a  situation  of  neglect  within  the  acquisition  workforce  in  a  time  of  persistent 
conflict  when  the  workforce  could  have  been  utilized  to  its  fullest,  ultimately  causing 
the  workforce  to  break  down  under  the  strain  of  too  many  requirements  and  too  few 
trained  personnel. 

2.  Defense  Acquisition  Workforce  Improvement  Act 

As  a  result  of  an  acquisition  workforce  that  was  plagued  by  scandal  and 
inefficiencies,  the  DAWIA  was  developed  and  enacted  into  law  in  1990.  Fraud, 
waste,  and  abuse  have  gone  hand  in  hand  with  military  operations  throughout 
history.  The  U.S.  government’s  response  after  each  conflict  has  been  to  enact 
progressively  more  restrictive  legislation  in  the  name  of  acquisition  reform.  Defense 
historian  William  Gregory  (1989)  described  in  his  work  The  Defense  Procurement 
Mess  the  state  of  acquisitions  in  the  late  1980s  “as  one  that  had  been  managed  and 
over-reformed  into  impotence  with  volumes  of  oversight  regulations.”  Scandals 
dragged  the  issues  further  into  the  spotlight.  Operation  III  Wind,  a  Department  of 
Justice  probe  that  uncovered  widespread  corruption  and  incidents  of  fraud  and 
bribery  within  the  defense  procurement  system,  was  concluded  in  1988.  In  all,  the 
operation  resulted  in  more  than  60  convictions,  including  that  of  former  Assistant 
Secretary  of  the  Navy  Meivyn  Paisley  (Layton,  2007). 
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Voted  into  law  as  Public  Law  101-510  on  November  5,  1990,  the  DAWIA  was 
codified  in  Title  10,  Chapter  87  of  the  U.S.  Code  and  amended  in  1991,  1993,  2001, 
2003,  2008,  2011,  and  2013  (DAWIA,  2013).  The  purpose  of  the  DAWIA  was  to  shift 
the  focus  from  regulating  acquisition  procedures  to  developing  a  professional 
acquisition  workforce.  It  established  a  certification  process  for  contracting  as  a 
career  field,  to  include  establishing  standards  for  education,  training,  and 
progressive  levels  of  experience.  DAWIA  also  prescribed  the  creation  of  a  center  for 
defense  acquisition  education.  The  Defense  Acquisition  University  (DAU)  was 
formed  in  1992  to  satisfy  that  requirement.  The  DAU  was  initially  established  from  a 
consortium  of  existing  DoD  organizations,  bound  together  by  memoranda  of 
agreement  to  cooperate  as  the  DAU.  The  DAU  has  been  highly  successful  in  raising 
professional  standards  of  education  for  acquisition  professionals;  however,  while 
quality  has  increased,  quantity  has  not  (Layton,  2007). 

3.  The  Acquisition  Workforce  in  Decline 

Much  has  been  written  over  the  past  six  years  about  issues  facing  the 
acquisition  workforce.  The  Acquisition  Advisory  Panel  (AAP)  issued  a  report  in  2006 
through  the  Office  of  Federal  Procurement  Policy  (OFPP)  detailing  the  challenges 
facing  the  acquisition  workforce.  The  acquisition  career  field  was  not  spared  from  the 
larger  defense  drawdown  of  the  early  to  mid-1990s.  Actual  numbers  for  the  time 
period  are  difficult  to  find,  for  two  main  reasons.  First,  each  organization  in  the 
federal  government  defines  “acquisition  professional”  differently;  and  second,  prior  to 
1999,  there  was  no  requirement  mandating  tracking  acquisition  professionals 
separately  within  the  federal  employment  system.  What  is  known  is  that  the 
acquisition  workforce  went  through  substantial  reductions  throughout  the  1990s,  and 
little  to  no  hiring  took  place.  Meanwhile,  the  workforce  continued  to  gain  experience 
while  it  edged  ever  closer  to  retirement  (Office  of  Federal  Procurement  Policy 
[OFPP],  2007).  Figure  2  illustrates  the  issue,  with  a  large  number  of  contracting 
officers  with  less  than  10  years  of  experience,  with  a  dip  in  the  mid-level  experience 
range,  followed  by  a  large  number  of  employees  at  the  retirement  age. 
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Figure  2.  Defense  Acquisition  Workforce  Retirement  Eligibility  Distribution 

(DoD,  2010,  p.  2-22) 

Since  1999,  the  differences  across  the  federal  government  in  defining 
acquisition  professionals  make  it  difficult  to  ascertain  precise  statistics  about  growth 
trends  within  the  workforce,  though  all  numbers  point  towards  decline,  ranging 
anywhere  from  3%  to  27%  (OFPP,  2007).  Within  the  current  acquisition  workforce, 
50%  were  retirement  eligible  in  2010  (Girovasi,  2007).  Another  well-documented  fact 
is  the  increase  in  acquisition-related  spending  since  1990.  Paradoxically,  as  the 
acquisition  workforce  was  reduced,  the  reliance  on  the  workforce  increased. 
Consider  some  examples: 

•  Federal  acquisition  expenditures  tripled  from  FY  1991  to  FY  2006, 
reaching  a  level  of  $424  billion. 

•  Federal  acquisition  spending  increased  by  65%  from  FY  2001  to  FY 
2005,  representing  an  increase  from  $235  billion  to  $388  billion. 

•  Within  the  DoD,  workload  per  GS-1 102  (Contract  Specialist)  staff 
member  rose  from  an  annual  average  of  $6.4  million  in  contract 
actions  in  FY  1996  to  nearly  $13  million  in  FY  2005. 

•  Contracts  for  services  accounted  for  60%  of  total  spending  in  FY  2005 
and  2006  (Girovasi,  2007). 
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The  issue  at  hand  becomes  one  of  more  contract  actions  to  process  with 
fewer,  less-experienced  contracting  officers  available  to  process  them.  The  2007 
Gansler  Commission  report  on  Army  contracting  noted  that  over  a  12-year  period, 
contracting  actions  had  risen  350%  while  the  Army  contracting  workforce  had  been 
reduced  by  50%  (Gansler,  2007). 

4.  The  System  Breakdown 

Several  reports,  documents,  and  commissioned  studies  have  established 
what  happened  once  the  contracting  system  was  placed  under  stress.  GAO-02-737 
was  issued  in  July  2002,  approximately  10  years  after  the  overhaul  of  the  acquisition 
system  began,  but  before  the  United  States’  involvement  in  Iraq  and  Afghanistan 
and  the  surge  in  contracting  actions.  The  GAO  (2002)  noted  that  when  the  DoD  had 
adopted  a  definition  of  the  acquisition  workforce,  it  was  very  multidisciplinary  in 
nature,  covering  various  functions  of  the  contracting  team.  Various  government 
civilian  agencies,  by  contrast,  had  not  taken  such  a  broad  approach  and  only 
considered  the  GS-1 102  series  to  make  up  the  contracting  team.  As  a  result  of  the 
DoD  outlook,  they  had  developed  broad  training  and  tracking  programs  to  assist  in 
maintaining  and  accounting  for  their  acquisition  professionals.  The  GAO  (2002) 
found  overall  that  the  DoD  was  executing  the  DAWIA  reforms  well;  however,  the 
government  civilian  agencies  still  had  work  to  do.  The  report  also  found,  through  the 
interview  process,  that  acquisition  leadership  felt  that  funding  was  currently 
adequate  to  maintain  training  proficiency  levels,  but  that  leadership  worried  about 
upcoming  budget  cuts,  particularly  to  the  DAU’s  budget  (GAO,  2002). 

By  2005,  the  situation  seemed  to  have  worsened.  GAO-05-274,  Contract 
Management:  Opportunities  to  Improve  Surveillance  on  DoD  Service  Contracts, 
issued  in  March  2005,  reviewed  90  service  contracts.  Out  of  these  90  contracts,  the 
GAO  found  insufficient  surveillance  on  24.  From  those  24  contracts,  15  of  them  had 
no  surveillance  whatsoever.  The  DoD  readily  acknowledged  the  lapse,  and  in 
response  to  the  findings  stated  they  simply  did  not  have  enough  qualified  contracting 
personnel  to  fulfil  the  requirement  (GAO,  2005). 

Testimony  by  David  M.  Walker,  Comptroller  General  of  the  U.S.,  before  the 
House  of  Representatives  Subcommittee  on  Defense  and  the  Committee  on 
Appropriations  in  September  2006,  was  captured  in  GAO-06-800T  (Walker,  2006). 
His  testimony  was  based  on  six  years  of  research  using  the  generally  accepted 
government  auditing  standards  (GAGAS).  The  work  revealed  that  dollar  values  for 
major  weapon  system  and  service  contracts  were  indeed  on  a  sharp  rise,  and  the 
workforce  had  remained  flat.  Issues  of  insufficient  oversight  of  contractors,  weak 
business  practices,  and  poor  incentives  for  contractors  to  perform  well  were  also 
identified  (Walker,  2006).  Figure  3  illustrates  the  issue;  the  line  within  the  figure 
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demonstrates  the  acquisition  workforce,  while  the  columns  represent  the  increase  in 
obligations. 


#  GAO 

DOD  Contract  Obligations  and  Acquisition  Workforce 
Trends  Since  Fiscal  Year  2000 
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Figure  3.  DoD  Contract  Obligations  and  Acquisition  Workforce  Trends 

(GAO,  2006,  p.  4) 

In  late  2006,  the  GAO  also  convened  a  forum  of  experts  on  acquisition  and 
management  from  government,  academia,  and  the  public  sector  to  discuss  overall 
federal  acquisition  challenges  over  the  longer  term.  GAO-07-45SP  found  that 
leaders  among  the  various  federal  agencies  were  not  recognizing  the  important  role 
the  acquisition  workforce  played  in  their  organizations.  It  also  upheld  DoD  contract 
management  as  a  federal  government  high  risk  area,  a  status  it  had  held  since  1992 
(GAO,  2006). 

As  stories  of  fraud,  waste,  and  abuse,  coupled  with  rampant  spending  in  Iraq, 
Afghanistan,  and  Kuwait,  continued  to  mount,  the  secretary  of  the  Army  established 
a  commission  to  examine  expeditionary  contracting  in  2007.  The  commission  was 
headed  by  Dr.  Jacques  S.  Gansler,  former  under  secretary  of  defense  for 
acquisition,  technology,  &  logistics.  The  Gansler  Commission  found,  as  several 
previous  GAO  reports  show,  that  the  contracting  workforce  had  become  short- 
staffed  and  undertrained.  It  also  highlighted  massive  amounts  of  fraud  relating  to 
government  contracts,  mostly  in  Iraq.  As  of  October  23,  2007,  83  Army  criminal 
investigations  relating  to  contract  fraud  were  ongoing  in  Iraq,  Afghanistan,  and 
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Kuwait.  At  the  same  time,  23  government  employees  (both  civilian  and  military)  were 
charged  or  indicted  in  federal  court.  The  contracts  affected  represented  over  $6 
billion  in  value  (Gansler,  2007). 

Further  testimony  before  Congress  by  David  M.  Walker  was  captured  in  two 
additional  GAO  reports,  GAO-07-1098T  and  GAO-08-621T  (Walker,  2007,  2008). 
These  reports  continued  to  highlight  issues  of  shortages  in  the  workforce  and 
insufficiently  trained  acquisition  personnel.  Additionally,  one  report  noted  the 
dramatic  increase  in  the  role  of  service  contracts  within  the  DoD  and  a  shift  away 
from  the  utilization  of  government  employees.  Concerns  were  also  raised  about  the 
line  becoming  blurred  between  contractor  involvement  with  inherently  governmental 
functions.  The  marked  increase  in  the  number  of  service  contracts  only  multiplied 
the  stress  on  the  government  contracting  workforce  (GAO,  2008).  A  report  delivered 
by  John  P.  Hutton,  Director  of  Acquisition  and  Sourcing  Management,  to  Congress 
in  2007  reinforced  this  by  revealing  that  the  DoD  had  awarded  service  contracts  for 
security  guard  services  at  57  domestic  bases,  and  that  46  of  them  were  awarded  on 
a  sole  source  basis.  The  cost  of  the  contracts  rose  by  25%  compared  to  when  the 
same  contracts  were  previously  awarded  competitively.  The  same  report  identified 
that,  indexed  to  2006  dollars,  the  amount  awarded  for  DoD  service  contracts  rose 
from  $85.1  billion  in  1996  to  $151  billion  in  2006,  representing  a  78%  increase 
(Hutton,  2007). 

Of  specific  mention,  the  GAO  issued  the  report  titled  Status  of  DoD’ s 
Implementation  of  Independent  Management  Reviews  for  Service  Acquisitions  in 
January  2010.  Overall,  the  report  found  that  there  are  still  issues  with  the 
implementation,  but  the  DoD  is  making  progress.  The  report  noted  two  major  issues. 
First,  at  the  time  of  the  report,  the  military  departments  stated  that  they  had 
undertaken  hundreds  of  reviews;  however,  they  could  not  deliver  precise  numbers 
because  of  a  lack  of  a  defined  reporting  process.  Second,  and  perhaps  equally 
grievous,  the  DoD  had  not  yet  developed  a  methodology  or  mechanism  to  report 
review  results  and  lessons  learned  back  to  the  force  (GAO,  2010).  The  following 
section  addresses  the  development  of  the  peer  review-requirement. 

C.  DEVELOPMENT  OF  PEER  REVIEW  REQUIREMENT 

The  development  of  the  peer-review  requirement  began  in  2008  with  a  DoD 
memorandum  informing  the  service  branches  that  contractual  actions  over  a  certain 
dollar  threshold  would  be  subject  to  a  DoD-level  peer-review  process,  both  pre-  and 
post-award  of  the  contract.  It  also  instructed  the  respective  services  to  establish 
internal  peer-review  processes  based  on  lower  dollar  thresholds.  The  Defense 
Federal  Acquisition  Regulation  Supplement  (DFARS)  and  Army  Federal  Acquisition 
Regulation  Supplement  (AFARS;  2014)  provided  specific  guidance  about  the 
implementation  of  the  new  policy,  and  contracting-specific  guides  such  as  the 
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Contract  Attorney’s  Deskbook  (Contract  and  Fiscal  Law  Department,  Judge 
Advocate  General’s  School,  2013)  and  the  Army  Contracting  Command  Desk  Book 
(Army  Contracting  Command  [ACC],  2012)  provided  references  to  the  new 
regulations. 

1 .  DoD  Peer-Review  Implementation  Memorandum 

The  DPAP  organization  is  the  policy  arm  of  DoD  for  defense  procurement  and 
acquisitions.  This  organization  is  responsible  for  “all  Contracting  and  Procurement 
policy  matters  including  e-Business  in  the  DoD.  DPAP  executes  that  policy  through 
the  timely  update  of  the  DFARS,  PGI,  and  5000.1&2’’  (DPAP,  2014).  The  DoD 
established  the  requirement  for  contract  peer  reviews  by  issuing  a  policy 
memorandum  dated  September  29,  2008,  and  titled  “Peer  Reviews  of  Contracts  for 
Supplies  and  Services”  (Assad,  2008).  The  memorandum  stated  three  primary 
objectives:  (1 )  to  ensure  that  contracting  officers  across  the  Department  are 
implementing  policy  and  regulations  in  a  consistent  and  appropriate  manner;  (2)  to 
continue  to  improve  the  quality  of  contracting  processes  across  the  Department;  (3) 
to  facilitate  cross-sharing  of  best  practices  and  lessons  learned  across  the 
Department  (Assad,  2008). 

The  peer-review  requirement  grew  from  two  sources.  First,  the  DoD  had 
implemented  a  pilot  program  to  evaluate  proposed  contract  awards,  focusing  on  the 
pre-contract  award  process.  Second,  Congress  enacted  Section  808  of  the  National 
Defense  Authorization  Act  (NDAA)  for  Fiscal  Year  2008,  Public  Law  110-181 .  This 
law  required  the  DoD  to  establish  requirements  for  post-award  independent 
management  reviews  of  contracts  for  services,  and  for  sharing  lessons  learned  from 
those  reviews  (NDAA,  2007).  The  existing  peer-review  pilot  program  expanded  to 
satisfy  the  requirement  brought  about  by  the  NDAA  of  2008. 

Rather  than  being  regulatory  in  nature,  the  peer  reviews  were  envisioned  as  a 
quality  control  and  an  advisory  tool  for  contracting  officers.  It  is  important  to  note  that 
the  agency  managing  the  procurement  still  has  the  final  decision  on  how  the 
procurement  is  executed,  though  the  memorandum  did  establish  a  requirement  that 
all  peer-review  recommendations  and  their  dispositions  be  documented  in  the 
contract  file. 

The  DPAP  office  has  responsibility  for  organizing  review  teams  and 
facilitating  pre-award  peer  reviews  for  all  contracts  valued  at  or  above  $1  billion,  and 
post-award  reviews  of  service  contracts  valued  at  or  above  $1  billion.  The  contract 
value  should  also  include  the  estimated  value  of  any  options  associated  with  the 
contract.  The  DPAP-level  peer-review  teams  consist  of  senior-level  DoD  contracting 
officials,  both  civilian  and  military,  and  members  of  the  Office  of  General  Counsel. 
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Further,  the  initial  memorandum  directed  that  each  military  department, 
defense  agency,  and  DoD  field  activity  would  publish  its  own  policies  for  the  conduct 
of  pre-  and  post-award  peer  reviews  for  contracts  valued  at  less  than  $1  billion. 

2.  Defense  Federal  Acquisition  Regulation  Supplement  Peer-Review 
Requirements 

The  DoD  codified  the  requirement  into  the  Defense  supplement  to  the  Federal 
Acquisition  Regulation  (FAR).  The  FAR  is  a  detailed  federal  regulation  that 
prescribes  how  the  federal  government  will  procure  supplies,  services,  and 
equipment.  The  DoD  has  its  own  supplement  to  the  FAR,  as  does  each  service 
component  (Army,  Navy,  and  Air  Force).  DFARS  Part  201.170  explains  the 
requirement  for  peer  reviews: 

201.170  Peer  Reviews. 

(a)  DoD  peer  reviews. 

(1)  The  Office  of  the  Director,  Defense  Procurement  and 
Acquisition  Policy,  will  organize  teams  of  reviewers  and  facilitate 
peer  reviews  for  solicitations  and  contracts,  as  follows  using  the 
procedures  at  PGI  201.170 — 

(i)  Pre-award  peer  reviews  for  competitive  procurements  will  be  conducted  in  three 
phases  for  all  solicitations  valued  at  $1  billion  or  more; 

(ii)  Pre-award  peer  reviews  for  noncompetitive 
procurements  will  be  conducted  in  two  phases  for  new 
contract  actions  valued  at  $500  million  or  more;  and 

(iii)  Post-award  peer  reviews  will  be  conducted  for  all 
contracts  for  services  valued  at  $1  billion  or  more. 

(2)  To  facilitate  planning  for  peer  reviews,  the  military 
departments  and  defense  agencies  shall  provide  a  rolling 
annual  forecast  of  acquisitions  that  will  be  subject  to  DoD  peer 
reviews  at  the  end  of  each  quarter  (i.e.,  March  31 ;  June  30; 
September  30;  December  31),  to  the  Deputy  Director,  Defense 
Procurement  and  Acquisition  Policy  (Contract  Policy  and 
International  Contracting)  via  e-mail  to  osd.pentaqon.ousd- 
atl.mbx.peer-reviews@mail. 

(b)  Component  peer  reviews.  The  military  departments  and  defense 
agencies  shall  establish  procedures  for — 

(1)  Pre-award  peer  reviews  of  solicitations  for  competitive 
procurements  valued  at  less  than  $1  billion; 

(2)  Pre-award  peer  reviews  for  noncompetitive  procurements 
valued  at  less  than  $500  million;  and 
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(3)  Post-award  peer  reviews  of  all  contracts  for  services  valued 
at  less  than  $1  billion.  (DFARS  201.170) 

The  DFARS  mentions  peer  reviews  in  Subpart  215.270,  Solicitation  and 
Receipt  of  Proposals  and  Information;  Subpart  207.104,  General  Procedures  of 
Acquisition  Planning;  and  Subpart  237.102,  Policy  of  Service  Contracts,  but  only  to 
the  extent  of  referring  back  to  Subpart  201.170.  The  FAR,  in  and  of  itself,  makes  no 
reference  to  peer  reviews.  Since  peer  reviews  are  a  DoD  requirement,  there  is  no 
regulatory  requirement  for  the  policy  on  peer  reviews  to  appear  in  the  FAR. 

3.  Overview  of  Contract  Attorney’s  Deskbook  Peer-Review 
Requirements 

The  2013  Contract  Attorney’s  Deskbook  (CADB),  published  by  the  Contract 
and  Fiscal  Law  Department  of  the  DoD’s  Judge  Advocate  General’s  School, 
highlights  the  need  for  peer  reviews.  Chapter  8,  Negotiated  Procurements  and 
Source  Selection,  mentions  fairly  early  that  peer  reviews  are  a  planning 
consideration  to  be  considered  in  the  acquisition  planning  phase,  and  reiterates  the 
requirements  stated  in  DFARS  201.170.  The  CADB  does  not  provide  any  more 
details  than  what  is  included  in  the  DFARS — only  that  it  is  a  statutory  requirement 
(Contract  and  Fiscal  Law  Department,  Judge  Advocate  General’s  School,  2013). 

The  next  section  examines  the  procedures,  guidance,  and  information  (PGI)  relating 
to  the  peer-review  requirement. 

D.  PROCEDURES,  GUIDANCE,  AND  INFORMATION  201.170  CONDUCTING 
PEER  REVIEWS 

The  DFARS  (2014)  contains  implementation  and  DoD-specific 
supplementation  information  to  the  FAR  (2014).  The  DFARS  sets  forth  requirements 
established  by  law,  policies  that  are  implemented  across  the  DoD,  delegation 
authority,  DoD-specific  FAR  deviations,  and  other  policies  deemed  to  have  a 
significant  impact  on  the  public.  The  PGI  series  offers  supplementation  to  the 
DFARS.  The  PGI  contains  techniques,  procedures,  and  guidance  of  a  non- 
regulatory  nature  that  are  not  included  in  the  DFARS  (DFARS  201.170). 

PGI  201 .170,  Conducting  Peer  Reviews,  provides  guidance  on  how  to 
conduct  peer  reviews  in  accordance  with  DFARS  201.170.  Since  the  PGI  201.170  is 
issued  as  guidance,  rather  than  strict  regulation,  it  provides  a  view  into  what  the 
DPAP  considers  most  important  within  the  peer-review  process.  Consider  the 
longest  paragraph  in  the  section: 

The  results  and  recommendations  that  are  products  of  peer  reviews 
are  intended  to  be  advisory  in  nature;  however,  in  the  event  the  peer 
review  report  includes  a  recommendation  that  is  identified  as 
“significant”  and  the  contracting  officer  does  not  intend  to  follow  that 
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recommendation,  the  senior  procurement  official  of  the  contracting 
activity  for  the  reviewed  organization  must  be  made  aware  of  this  fact 
before  action  is  taken  (or  inaction,  as  applicable)  that  is  contrary  to  the 
recommendation.  Reviews  will  be  conducted  in  a  manner  that 
preserves  the  authority,  judgment,  and  discretion  of  the  contracting 
officer  and  the  senior  officials  of  the  acquiring  activity.  (DFARS 
201.170) 


This  section  would  seem  to  indicate  the  importance  placed  on  the  peer-review 
results.  It  also  highlights  an  interesting  dichotomy  between  the  sanctity  of  the 
contracting  officer  and  the  recommendations  of  the  peer-review  board.  It  is  explicit  in 
this  statement  that  the  contracting  officer  can  stand  by  his  or  her  decision,  though  it 
will  be  under  the  spotlight. 

PGI  201 .170  begins  by  stating  the  review  criteria  that  are  the  tenets  of  DoD- 
wide  contractual  actions,  and  includes  a  document  highlighting  these  areas.  This  is 
the  core  of  the  peer-review  process  and  the  benchmark  against  which  peer  reviews 
are  conducted.  Figures  4  and  5  describe  what  participants  in  the  peer-review 
process  are  examining  for  pre-  and  post-award  acquisition  of  services. 
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DoT)  Ciiteiirt  for  the  Acquisition  of  Sei^ices  (Pi  einvui  d) 


Figure  4.  DoD  Peer-Review  Criteria  for  Services  Acquisitions  During  the 

Pre-Award  Phase 

(DFARS) 
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DoD  Ciiteiirt  for  the  Arqiiisitioii  of  SeiAicos'  (Postsnvjii  <1) 


Figure  5.  DoD  Peer-Review  Criteria  for  Services  Acquisitions  During  the 

Post-Award  Phase 

(DFARS  237.102.76) 
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The  objectives  of  the  peer-review  process  are  simple:  to  ensure  that  DoD 
contracting  officers  are  implementing  policy  and  regulations  appropriately,  to 
improve  contracting  as  a  process  throughout  the  DoD,  and  to  facilitate  the  sharing  of 
ideas  and  best  practices  throughout  the  DoD.  Additionally,  the  DPAP  maintains  a 
database  of  peer-review  results  for  contracts  reviewed  at  the  DoD  level,  located  at 
www.acq.osd.mil/dpap/cpic/cp/peer_reviews.html,  which  is  the  subject  of  further 
analysis  in  this  thesis  (DPAP,  2013). 

According  to  PGI  201.170,  pre-award  peer-review  requirements  are 
separated  into  two  categories:  competitive  and  noncompetitive  acquisitions.  Pre¬ 
award  peer  reviews  for  competitive  acquisitions  are  required  at  three  different  points: 
issuing  the  solicitation,  requesting  the  final  proposal  revisions,  and  issuing  the 
contract  award.  For  noncompetitive  acquisitions,  the  requirement  is  for  peer  reviews 
at  negotiation  and  contract  award.  For  both  competitive  and  non-competitive,  post¬ 
award  peer  reviews  concentrate  on  adequacy  of  competition,  an  assessment  of 
contractor  performance,  and  how  adequate  the  government  surveillance  plan  of 
contractor  performance  has  proven  to  be  (DFARS  201 .170). 

PGI  201 .170  also  defines  the  peer-review  team: 

A  senior  official  designated  by  the  OSD  [Office  of  the  Secretary  of 
Defense]  Office  of  Small  Business  Programs  will  participate  as  a  team 
member  on  peer  reviews  of  services  acquisitions.  Teams  will  include 
civilian  employees  or  military  personnel  external  to  the  department, 
agency,  or  component  that  is  the  subject  of  the  peer  review.  (DFARS 
201.170) 

This  PGI  highlights  the  documents  that  peer-review  teams  must  be  able  to 
access.  Interestingly,  there  are  also  sections  in  the  PGI  that  read  like  best  practices 
for  both  the  pre-  and  post-award  phases: 

Pre-Award  Elements  to  be  Addressed 

•  The  process  was  well  understood  by  both  government  and  Industry; 

•  Source  Selection  was  carried  out  in  accordance  with  the  Source 
Selection  Plan  and  RFP; 

•  The  Source  Selection  Evaluation  Board  (SSEB)  evaluation  was  clearly 
documented; 

•  The  Source  Selection  Advisory  Council  (SSAC)  advisory  panel 
recommendation  was  clearly  documented; 

•  The  Source  Selection  Authority  (SSA)  decision  was  clearly  derived 
from  the  conduct  of  the  source  selection  process; 
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•  All  source  selection  documentation  is  consistent  with  the  Section  M 
evaluation  criteria;  and 

•  The  business  arrangement. 

Post-Award  Elements  to  be  Addressed 

•  Contract  performance  in  terms  of  cost,  schedule,  and  requirements; 

•  Use  of  contracting  mechanisms,  including  the  use  of  competition,  the 
contract  structure  and  type,  the  definition  of  contract  requirements, 
cost  or  pricing  methods,  the  award  and  negotiation  of  task  orders,  and 
management  and  oversight  mechanisms; 

•  Contractor’s  use,  management,  and  oversight  of  subcontractors; 

•  Staffing  of  contract  management  and  oversight  functions; 

•  Extent  of  any  pass-throughs,  and  excessive  pass-through  charges  by 
the  contractor  (as  defined  in  section  852  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2007,  Public  Law  109-364);  and 

•  Steps  taken  to  mitigate  the  risk  that,  as  implemented  and  administered, 
non-personal  services  contracts  may  become  de  facto  personal 
services  contracts.  (PGI  201.170) 

The  following  section  discusses  how  corporations  review  procurement  actions 
versus  how  the  DoD  examines  its  procurement  actions  in  the  form  of  the  peer-review 
process  for  contractual  actions. 

E.  CORPORATE  PROCUREMENT  MANAGEMENT,  FEDERAL 
GOVERNMENT  CONTRACTING,  AND  DOD  CONTRACTING:  A  BRIEF 
COMPARISON 

Initially,  a  quick  comparison  of  procurement  in  the  corporate  world  against 
contracting  within  the  federal  government  might  lead  the  uninitiated  to  think  they 
were  similar.  After  all,  both  follow  the  same  six-step  process:  procurement  planning, 
solicitation  planning,  solicitation,  source  selection,  contract  administration,  and 
contract  closeout  or  termination  (Garrett,  2007,  p.  21).  Under  more  detailed 
inspection,  vast  differences  within  the  motivation  of  the  parties  involved  and  the 
forces  that  shape  the  process  are  exposed.  Kovacs  (2004)  noted  in  his  work 
Enhancing  Procurement  Practices, 

The  basic  procurement  principles  and  techniques  are  equally 
applicable  in  both  the  public  and  private  sectors  ...  which  means:  with 
the  right  quality,  functionality  and  performance;  under  the  right  terms 
and  conditions,  among  which  costs  and  timely  implementation  usually 
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are  of  essence;  and,  with  the  right  long-term  operation  and/or 
maintenance  support,  (p.  175) 

The  divergence  begins  with  money  in  the  form  of  funding  and  the  method  by 
which  the  government  sector  and  private  sector  come  by  the  funding. 

Corporations  gain  funding  by  financing  debt  or  by  spending  their  own  profit 
that  they  have  retained  in  the  corporation.  In  either  instance,  they  are  motivated  by 
the  need  and  desire  to  utilize  those  funds  as  efficiently  as  possible  to  earn  the 
maximum  profit  possible  from  each  dollar  spent.  Thus,  there  is  no  need  for  a 
regulatory  requirement  within  the  overarching  corporate  world  for  a  formal  peer- 
review  process  of  contracts.  As  long  as  free  market  forces  are  in  play,  each 
corporation  is  driven  by  competition  to  be  as  efficient  as  possible  (Kovacs,  2004). 

The  chasm  widens  when  considering  business  relationships  and  how  they 
integrate  with  procurement  in  the  corporate  arena  versus  contracting  in  the 
government  sector.  Procurement  actions  outside  of  the  government  are  typically 
more  focused  on  the  business  relationship  and  building  and  maintaining  a 
partnership  for  future  ventures.  These  are  the  incentives,  intangible  and  tangible, 
such  as  more  favorable  credit  terms  on  purchases,  that  drive  corporations  to  follow 
through  on  the  whole  procurement  process.  Hence,  they  are  less  apt  to  neglect  the 
contract  administration  and  contract  closeout/  termination  phases  of  the 
procurement  process  over  federal  contracting  activities. 

Corporations  do  have  methods  for  evaluating,  or  perhaps  more  accurately, 
examining  how  they  are  performing.  Consulting  services  are  often  hired  from 
sources  outside  of  the  corporation  (or  in  cases  of  extremely  large  corporations,  they 
may  be  contained  in-house  under  a  cloistered  division).  Consultants  execute 
functions  similar  to  what  a  peer  review  accomplishes,  in  that  someone  with  an  equal 
or  greater  amount  of  knowledge  on  the  topic  examines  a  subject  (Garrett,  2007). 

The  difference  lies  in  the  fact  that,  with  the  exception  of  extremely  large  contracts, 
consultant  actions  typically  focus  on  a  process,  with  an  expected  end  state  of  further 
optimizing  that  process.  Peer  reviews  of  DoD  contracts  are  conducted  once  Pre 
Award  and  once  Post  Award;  as  such,  the  peer-review  process  is  evaluating  an  end 
product  rather  than  the  process  that  delivered  the  product. 

There  is  a  tool  that  is  designed  to  help  both  buyers  and  sellers  evaluate  risk 
and  reward  associated  with  procurements.  “The  Contract  Management  Risk  and 
Opportunity  Assessments  Tool  (CMROAT)  is  designed  to  help  organizations,  both 
buyers  and  sellers,  assess  the  risk  and  opportunities  associated  with  a 
pending/potential  or  actual  contract”  (Garrett,  2007,  p.  234).  The  CMROAT  has 
developed  into  a  tool  utilized  by  both  government  and  private  entities;  however,  it 
focuses  on  the  risks  and  opportunities  associated  with  the  business  dealings  of  the 
contract  and  has  less  to  do  with  the  structure  and  content  of  the  contract  itself. 
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Overall,  the  federal  contracting  process  is  process-driven;  if  the  process  is 
followed  correctly,  then  the  end  product  is  typically  in  the  acceptable  range. 
Alternatively,  as  noted,  the  corporate  procurement  system  is  fueled  by  incentives.  As 
a  result,  corporate  review  mechanisms  tend  to  focus  on  optimizing  a  process  for  a 
greater  return  on  investment,  and  the  peer-review  process  as  it  applies  to  DoD 
contractual  actions  focuses  on  fixing  an  output  of  a  process  to  bring  it  in  line  with 
what  is  deemed  acceptable. 

F.  SUMMARY 

In  the  literature  review,  I  mentioned  several  aspects  of  the  acquisition  system 
and  how  they  contributed  to  the  peer-review  requirement.  With  the  implementation  of 
the  DAWIA,  the  DoD  made  significant  strides  in  developing  a  professional 
acquisition  workforce.  Negligence  of  the  workforce  throughout  the  late  1990s  and 
into  the  2000s  led  to  a  shortage  within  the  acquisition  workforce,  followed  by  a  surge 
in  acquisition  activity  accompanying  the  rapid  increase  in  overseas  contingency 
operations.  With  the  contracting  system  thoroughly  stressed,  numerous  examples  of 
inefficiencies,  fraud,  waste,  and  abuse  were  uncovered.  This  led  to  an  official  peer- 
review  requirement  from  Congress.  The  actual  peer-review  process  was  developed 
based  on  an  existing  pilot  review  program  and  expanded  to  meet  the  congressional 
requirement.  The  following  chapter  discusses  the  research  methodology. 
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III.  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  addresses  how  I  collected  the  data  and  used  the  analytics.  It 
also  includes  a  summary  of  the  DPAP  database  of  peer-review  results  to  provide 
context  for  the  data  analysis  chapter  of  my  research.  The  various  aspects  of  the 
DPAP  database  of  peer-review  results  are  also  described.  The  purpose  of  my 
research  was  not  to  provide  an  exhaustive  answer  to  an  in-depth  research  question; 
rather,  the  purpose  was  to  explore  a  previously  uncharted  research  area  to  examine 
whether  it  merited  further  study.  The  DPAP  database  of  peer-review  results  was 
broken  down  categorically  by  the  various  data  elements  of  data  present  and 
analyzed  quantitatively  based  on  frequency  of  occurrences  of  a  particular  data 
element. 

B.  DPAP  DATABASE  OF  PEER  REVIEW  RESULTS  OVERVIEW 

The  DPAP  database  of  peer-review  results  resides  in  the  contract  policy 
section  of  the  DPAP  website,  in  both  PDF  and  Excel  versions.  The  purpose  of  the 
DPAP  database  of  peer-review  results  is  to  collect  the  results  from  peer  reviews  of 
contracts  that  met  the  threshold  ($1  billion  and  above)  for  DoD-level  peer  review. 
According  to  the  Peer  Review  section  of  the  DPAP  website, 

The  Peer  Review  program  improves  the  quality  of  the  Department’s 
contracting  processes  by  sharing  lessons  learned  and  best  practices 
and  ensuring  that  contracting  officers  implement  policy  and  regulations 
in  a  consistent  manner.  The  program  is  implemented  in  accordance 
with  DODI  5000.02  and  DFARS  Part  201.  (DPAP,  2014) 

1 .  Summary  of  the  Parts  of  the  DPAP  Database  of  Peer-Review 
Results 

The  DPAP  database  of  peer-review  results  itself  is  organized  in  a  simple 
format.  Figure  6  shows  an  example  page  from  the  DPAP  database  of  peer-review 
results  as  of  December  20,  2013. 
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Figure  6.  Excerpt  From  DPAP  Database  for  OSD-Level  Peer  Reviews 

(DPAP,  2013) 

Each  row  consists  of  an  entry  for  a  particular  data  element  of  a  contract,  and 
the  columns  contain  the  appropriate  type  of  data  related  to  that  contract  entry.  The 
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DPAP  database  of  peer-review  results  has  been  anonymized,  so  there  is  no  way  to 
discern  how  many  entries  apply  to  the  same  contract.  The  Category  column  lists  the 
category  the  peer-review  comment  falls  within.  There  are  eight  categories:  Incentive 
and  Award  Fee,  Market  Research,  Peer  Review,  Post-Award  Administration,  Pricing, 
Requirements/Performance  Work  Statement/Scope  of  Work,  Source  Selection,  and 
Terms  and  Conditions.  The  Types  of  Contracts  column  spans  a  staggering  31 
different  types,  consisting  of  competitive  and  non-competitive,  services,  goods,  and 
construction  contracts. 

The  Feedback  column  provides  a  simple  narrative  comment  regarding  the 
opinion  of  the  peer-review  team.  The  column  entitled  Review  Phase  illustrates  the 
phase  of  the  contracting  process.  There  are  currently  six  phases:  Phase  1 ,  Phase  2, 
Phase  2&3,  Phase  3,  Phase  4,  and  Post  Award  1 .  The  final  column  relates  to  the 
Type  of  Feedback  provided,  and  consists  of  three  categories:  Recommendation, 
Best  Practice,  and  Lessons  Learned.  Figure  6  summarizes  the  different  database 
fields. 


2.  Phases  of  the  Contracting  Process  Versus  DPAP  Peer-Review 
Phases 

The  DPAP  database  of  peer-review  results  provides  no  explanation  about  the 
timing  of  the  Review  Phases  used,  nor  do  the  Frequently  Asked  Questions  or 
Standard  Operating  Procedures  for  peer  reviews  on  DPAP’s  website.  Further 
research  uncovered  an  explanation  of  four  phases  in  the  GAO  (2010)  report  Status 
of  Do D’s  Implementation  of  Independent  Management  Reviews  for  Services 
Acquisitions.  Phase  1  is  prior  to  the  issuance  of  the  solicitation.  The  documents 
reviewed  consist  of  the  performance  work  statement,  quality  assurance  surveillance 
plan,  request  for  proposal,  and  source  selection  plan.  Phase  2  is  prior  to  the  request 
for  final  proposal  revisions.  The  documents  reviewed  are  instructions  for  proposals 
and  proposal  evaluation  criteria,  source  selection  evaluation  guide,  source  selection 
plan,  and  evaluations  of  contractor  proposals.  Phase  3  is  prior  to  contract  award. 

The  documents  reviewed  consist  of  the  proposal-analysis  report  and  selection- 
decision  document.  Phase  4  is  the  post-award  review.  The  documents  reviewed 
consist  of  any  documentation  related  to  the  program,  such  as  task  orders,  award-fee 
plan,  and  performance  assessments  (GAO,  2010,  p.  6).  Figure  7  highlights  the 
comparison  between  the  DPAP  Peer  Review  Phases  and  phases  of  the  contracting 
process. 
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Review  Phases 

1)  Prior  to  issuance  of  the  solieitation 

(No  eonparable 
step) 

2)  Prior  to  request  for  final  proposal 
revisions  &  3)  Prior  to  eontraet  award 

4)  Post-award  review 

Contracting 
Process  Phases 

1)  Proeurement 
Planning 

2)  Solieitation 
Planning 

3)  Solieitation 

4)  Souree  Seleetion 

5)  Contraet 
Administration 

6)  Contraet 
Closeout 

Figure  7.  DPAP  Peer-Review  Phases  Compared  to  the  Contracting  Process 

Phases 

The  DPAP  database  of  peer-reviews  Review  Phases  actually  contain  Phases 
1  through  4,  and  an  additional  Post-Award  1  phase.  Analyzing  the  narrative 
feedback  seems  to  indicate  that  in  practice,  the  Phase  4  review  described  in  the 
aforementioned  GAO  (2010)  report  occurs  shortly  after  contract  award  and  the  Post- 
Award  1  review  occurs  well  after  contractor  performance  is  underway. 

C.  TYPES  OF  DATA  ANALYSIS 

1.  DPAP  Database  of  Peer-Review  Results  Data  Element  Fields 

Within  the  DPAP  database  of  peer-review  results,  there  are  288  records. 

Each  data  element  was  subdivided  within  the  spreadsheet  to  allow  for  the 
identification  of  the  frequency  of  occurrence  for  each  data  element.  These  data  were 
then  used  to  identify  trends  within  the  DPAP  database  of  peer-review  results. 
Descriptive  analysis  of  the  frequency  of  observations  for  each  category  of  feedback 
was  conducted. 

The  eight  Categories,  31  Types  of  Contracts,  six  Review  Phases,  and  three 
Types  of  Feedback  were  individually  extracted  from  the  DPAP  database  of  peer- 
review  results.  The  results  of  each  data  element  were  tallied  and  recorded  and  then 
analyzed  by  frequency  of  occurrence  against  the  other  data  elements,  with  the  goal 
of  identifying  overarching  trends  in  the  data  elements. 

Since  the  preponderance  of  the  data  elements  were  contained  in 
Recommendation  Type  of  Feedback,  the  Category,  Type  of  Contract,  and  Review 
Phase  data  elements  were  further  analyzed  with  an  eye  towards  identifying  how 
much  Recommendation  Type  of  Feedback  appeared  in  each  of  those  data 
elements.  To  provide  additional  granularity,  occurrences  of  Recommendations  were 
expressed  as  percentage  of  the  total  occurrences  for  both  Categories  and  Contract 
Type  data  elements. 

Competency  gaps  identified  in  the  Department  of  Defense  Contracting 
Workforce  Competency  Assessment  Final  Report  (DPAP,  2007)  were  cross- 
referenced  against  the  DPAP  database  of  peer-review  results  to  identify  any  areas 
of  overlap  between  the  two.  Once  areas  of  overlap  had  been  identified,  occurrences 
of  Recommendation  Type  of  Feedback  for  the  overlap  areas  was  reviewed  to  see  if 
the  identified  competency  gaps  had  increased  occurrences  of  Recommendations. 
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Narrative  Feedback  Themes 


The  Feedback  section  for  each  entry  was  also  examined  to  identify  any 
common  themes  within  the  feedback  provided.  The  primary  goal  for  examining  the 
content  of  the  Feedback  section  was  to  identify  apparent  incongruities  between 
listed  categories,  such  as  feedback  type,  and  the  actual  narrative  posted  within  the 
Feedback  column.  My  examination  is  concerned  with  identifying  irregularities  with 
policy  and  regulations. 

D.  SUMMARY 

The  Methodology  chapter  provided  a  brief  overview  of  the  DPAP  database  of 
peer-review  results  and  how  it  was  subdivided  for  analysis.  The  Review  Phases 
were  compared  to  the  phases  of  the  contracting  process.  How  the  narrative 
Feedback  themes  were  reviewed  was  also  addressed.  The  next  chapter.  Chapter  IV, 
presents  the  results  of  the  data  analysis. 
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IV.  DATA  ANALYSIS 


A.  INTRODUCTION 

The  previous  chapter  discussed  how  the  data  were  analyzed.  This  section 
focuses  on  trends  within  the  DPAP  database  of  peer-review  results  by  various  data 
elements  of  the  data.  Frequencies  of  occurrences  by  Category,  Type  of  Contract, 
Review  Phase,  and  Type  of  Feedback  are  examined.  The  Recommendation  Type  of 
Feedback  receives  additional  review  because  of  its  predominance  of  occurrences 
within  the  DPAP  database  of  peer-review  results.  Relationships  between  the  DoD 
Contracting  Competency  Model  (CCM)  and  trends  within  the  DPAP  database  of 
peer-review  results  are  compared.  Finally,  themes  within  the  narrative  feedback 
section  are  also  investigated. 

B.  DESCRIPTIVE  STATISTICS 

The  various  data  elements  within  the  DPAP  database  of  peer-review  results 
noted  in  the  methodology  chapter  were  analyzed  for  frequency  of  occurrence.  The 
importance  of  breaking  down  the  DPAP  database  of  peer-review  results  in  this 
manner  lies  in  the  fact  that  it  allows  contracting  officers  and  policy-makers  to  identify 
areas  where  systemic  issues  within  the  contracting  process  persist.  The  data  on 
frequencies  of  occurrences  and  the  underlying  trends  can  then  be  used  to  target  the 
development  of  educational  and  training  programs  to  address  problem  areas.  A 
process  as  complex  as  developing  high  dollar  defense  contracts  may  well  never  be 
without  faults,  but  previous  deficiencies  need  to  be  addressed  to  avoid  future 
duplication  of  effort. 

1 .  Category 

Working  across  the  DPAP  database  of  peer-review  results,  the  first  data 
element  encountered  is  the  Category  column.  The  largest  category  was  the  source 
selection  category  at  115  of  288  observations,  which  should  not  be  surprising  given 
two  facts:  (1 )  the  emphasis  that  is  placed  on  the  source  selection  process  within 
DoD  contracting;  and  (2)  the  number  of  protests  that  originate  in  the  source  selection 
process,  stemming  from  allegations  of  issues  such  as  unequal  application  of 
evaluation  criteria,  unfair  negotiation  and  discussion  practices,  and  poor  use  of  Best 
Value  trade-offs  and  Lowest  Price  Technically  Acceptable  criteria  for  contract  award 
(White,  2012).  Figure  8  represents  the  occurrences  of  all  the  category  types 
represented  in  the  DPAP  database  of  peer-review  results. 
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Occurrences  by  Category  Type 
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■  Occurrences 


Figure  8.  Frequency  Distribution  of  Category  Type 

(Adapted  from  DPAP,  2013) 

The  second  cluster  of  occurrences  centered  on  Terms  and  Conditions, 

Pricing,  and  Post  Award  Administration.  It  is  notable  that  pricing,  while  the  second 
largest  category,  is  still  a  distant  second  with  46  occurrences.  Given  issues  that 
seem  to  pervade  the  pricing  realm  within  contracting,  it  would  seem  that  this  area 
would  receive  more  attention.  Bringing  the  point  into  sharper  focus  is  the 
consideration  that  this  data  represents  high  dollar  contracts  valued  at  $1  billion  and 
above,  and  that  high  value  defense  procurements  often  have  significant  cost 
overruns  exceeding  20%  of  the  original  cost  estimate  (GAO,  2014). 

2.  Type  of  Contract 

The  next  data  element  of  the  DPAP  database  of  peer-review  results  is  the 
Type  of  Contract  column.  The  contract  types  listed  comprise  a  staggering  31  types 
of  contracts.  The  most  frequently  occurring  type  is  the  Competitive  Multiple  Award 
Services  Contract  at  59  occurrences,  followed  by  the  Competitive  Services 
Contracts  and  Competitive  Multiple  Award  indefinite  delivery  indefinite  quantity 
(IDIQ)  at  39  and  20  occurrences,  respectively.  Figure  9  shows  the  various  types  of 
contracts  listed  and  the  number  of  times  the  occurrence  was  observed. 
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Peer  Review 

Market  Research 
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Occurrences  by  Contract  Type 
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Competitive  IDIQ  Services  Contract 
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Competitive  IDIQ 
Competitive  Design/Build  FPIF 
Commodity  (Competitive) 


I  Occurrences 


Figure  9.  Frequency  Distribution  of  Contract  Type 

(Adapted  from  DPAP,  2013) 

Admittedly,  the  terminology  of  some  of  the  listed  contract  types  could  be 
combined,  such  as  Competitive  Supplies  Contract  and  Competitive  Procurement  of 
Supplies.  The  issues  associated  with  having  such  a  broad  scope  of  contract  types 
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listed  is  addressed  in  a  subsequent  section  of  this  analysis  because  this  particular 
part  of  the  DPAP  database  of  peer-review  results  is  one  of  two  that  could  benefit  the 
most  from  refinement.  The  key  takeaway  from  this  section  of  data  appears  to  be  that 
overall,  the  DoD  is  peer  reviewing  more  contracts  for  services  than  anything  else. 
Service  contracts  comprise  over  65%,  or  189  occurrences,  of  the  Contract  Type 
entries.  This  is  not  surprising,  considering  the  amount  of  services  contracted  for 
within  the  DoD,  but  what  is  worth  considering  is  that  those  189  occurrences 
represent  a  dollar  value  of  at  least  $1  billion  each  (GAO,  2008). 

3.  Review  Phase 

The  next  data  element  to  address  is  the  Review  Phase  column.  Again,  one 
area  contains  the  majority  of  the  occurrences;  this  time  they  fall  into  Phase  1 ,  with  a 
total  of  157.  Phase  2  follows  with  64,  and  Phase  3  and  Post-Award  1  are  nearly  tied 
at  28  and  27  occurrences.  Figure  10  shows  the  occurrences  by  review  phase. 


Occurrences  by  Review  Phase 


■  Occurrences 


Figure  10.  Frequency  Distribution  of  Review  Phase 

(Adapted  from  DPAP,  2013) 

The  trends  demonstrated  within  the  occurrences  by  review  phase  would 
seem  to  indicate  that  learning  is  taking  place  as  the  contract  process  moves  along.  It 
also  seems  to  break  itself  naturally  between  Phase  4  and  Post-Award  1 ,  when  the 
contract  would  have  been  awarded  at  the  conclusion  of  the  source  selection  process 
and  contract  performance  had  begun.  This  action  would  move  the  contract  into  the 
contract  administration  portion  of  the  contracting  process.  Since  contract 
administration  represents  a  very  different  phase  of  the  contracting  process  from 
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procurement  planning,  solicitation  planning,  solicitation,  and  source  selection,  it 
follows  that  additional  learning  would  begin  to  occur  in  the  Post-Award  1  phase. 

4.  Type  of  Feedback 

The  final  quantifiable  data  element  of  the  DPAP  database  of  peer-review 
results  is  the  Type  of  Feedback  column.  This  column  seeks  to  assign  a  type  to  the 
narrative  feedback  column.  Consistent  with  the  other  DPAP  database  of  peer-review 
results  categories,  one  category  heavily  outweighed  the  others  with  occurrences. 
Recommendation  contained  190  occurrences,  with  Best  Practice  containing  68  and 
Lessons  Learned  amassing  30  occurrences.  Figure  1 1  demonstrates  the 
occurrences  by  Type  of  Feedback. 


Occurrences  by  Feedback  Type 


Lessons  Learned 


Best  Practice 


Recommendation 


■  Occurrences 


Figure  1 1 .  Frequency  Distribution  of  Type  of  Feedback 

(Adapted  from  DPAP,  2013) 


The  type  of  feedback  aspect  is  the  other  data  element  of  the  DPAP  database 
of  peer-review  results  that  could  benefit  from  additional  refinement.  The  Lessons 
Learned  Type  of  Feedback  ostensibly  provides  a  way  to  communicate  an  item  of 
information  the  peer-review  team  was  not  aware  of  to  the  readership  of  the  DPAP 
database  of  peer-review  results.  In  reading  the  narrative,  it  often  becomes  difficult  to 
distinguish  between  a  Lesson  Learned  and  a  Recommendation.  Best  Practices  are 
just  that — a  best  practice  that  the  peer-review  team  observed  from  the  organization 
or  contract  that  the  team  was  reviewing;  it  exists  as  a  way  to  propagate  those 
practices  back  out  to  the  contracting  workforce.  With  only  three  types  of  feedback 
categories,  and  since  most  of  the  occurrences  are  in  a  single  category,  it  is  difficult 
to  ascertain  anything  quantitatively  unless  the  recommendation  category  is  delved 
into  further,  which  occurs  in  the  next  section. 
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5.  Recommendation  Category 

The  Recommendation  category  contained  190  out  of  288  occurrences 
regarding  feedback  type.  That  comprises  65.97%  of  the  total  feedback  types.  It  is 
also  the  single  largest  density  of  occurrences  anywhere  in  the  DPAP  database  of 
peer-review  results.  As  such,  further  analysis  was  conducted. 

Examining  the  phases  revealed  that  most  Recommendations  were 
concentrated  in  Phase  1  reviews.  Phase  1  contained  130  Recommendation 
occurrences.  Phase  2  was  a  distant  second  at  38  occurrences,  with  12  occurrences 
in  Phase  3,  none  in  Phase  4,  and  1 1  in  Post-Award  1 .  Figure  12  elucidates  the 
recommendation  occurrences  by  phase. 


Recommendation  Occurrences  By  Phase 


Figure  12.  Frequency  Distribution  of  Recommendations  by  Phase 

(Adapted  from  DPAP,  2013) 

The  findings  are  consistent  with  the  idea  that  learning  is  taking  place 
throughout  the  contracting  process,  and  organizations  that  are  subject  to  formal 
review  by  a  peer-review  team  are  applying  the  observations  of  the  peer-review  team 
It  again  demonstrates  the  break  between  contract  award  and  contract  performance, 
illustrated  by  the  uptick  in  Recommendations  between  the  Phase  4  Review  Phase 
and  the  Post-Award  1  Review  Phase. 


Next,  the  number  of  occurrences  of  recommendations  within  each  Category 
was  examined.  The  categories  were  Terms  &  Conditions,  Source  Selection, 
Requirements/PWS/SOW,  Pricing,  Post  Award  Admin,  and  Incentive  and  Award 
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Fee.  Figure  13  exhibits  the  number  of  occurrences  of  Recommendations  within  each 
Category. 
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Figure  13.  Frequency  Distribution  of  Recommendations  Within  Categories 

(Adapted  from  DPAP,  2013) 

Another  important  way  to  view  this  data  is  by  percentage  of 
Recommendations  within  each  Category. 


Tabie  1 .  Percentage  of  Recommendations  by  Category 

(Adapted  from  DPAP,  2013) 


Categories 

Total 

Occurrences 

Occurrences  of 
Recommendation 

Percentage  of 
Recommendation 

Incentive  and  Award 
Fee 

23 

20 

86.96% 

Market  Research 

4 

0 

0% 

Peer  Review 

1 

0 

0% 

Post- Award  Admin 

35 

9 

25.71% 

Pricing 

46 

41 

89.13% 

Requirements/PWS/ 

SOW 

24 

17 

70.83% 

Source  Selection 

115 

66 

57.39% 

Terms  &  Conditions 

40 

37 

92.5% 

Total 

288 

190 

65.97% 

Table  1  lists  each  of  the  Category  types  in  the  DPAP  database  of  peer-review 
results.  Total  occurrences  is  the  number  of  times  that  Category  appears  within  the 
database.  Assuming  that  the  priority  of  the  DPAP  database  of  peer-review  results  is 
to  capture  issues  with  solicitations  and  contracts  to  assist  in  informing  the  DoD 
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contracting  workforce,  then  the  third  column  of  the  above  chart  is  the  most  important 
cross  section  of  the  data  to  consider.  It  details  the  number  of  Recommendations  per 
Category  within  the  DPAP  database  of  peer-review  results.  Consider  the  first  row  of 
Table  1 ;  it  expounds  on  the  Incentive  and  Award  Fee  Category.  That  Category 
appeared  23  times  in  the  DPAP  database  of  peer-review  results;  of  its  23 
appearances,  20  of  those  constituted  Recommendation  Feedback  Type,  with 
86.96%  of  the  occurrences  being  Recommendations.  The  percentage  column  is 
included  simply  to  help  visualize  the  Recommendation  occurrences  by  category. 

This  provides  a  more  precise  picture  within  the  DPAP  database  of  peer-review 
results  as  to  where  peer-review  teams  are  uncovering  issues.  If  one  examined  only 
the  occurrences  of  Recommendations,  the  conclusion  might  be  that  20 
Recommendations  might  not  be  significant,  but  once  the  dimension  is  added  that  20 
is  out  of  23,  and  86.96%  of  the  Feedback  Type  for  that  Category  is  a 
Recommendation,  then  there  may  be  an  issue  within  Incentive  and  Award  Fees  that 
needs  to  be  addressed.  By  scrutinizing  the  data  in  that  manner,  it  would  seem  the 
areas  of  concern  that  should  merit  additional  attention  are  Incentive  and  Award  Fee, 
Pricing,  Requirements/PWS/SOW,  and  Terms  &  Conditions.  Curiously  enough, 
while  Source  Selection  as  a  Category  is  the  most  frequently  occurring  Category 
within  the  DPAP  database  of  peer-review  results,  only  57.39%  of  its  appearances 
consist  of  Recommendations,  with  the  rest  comprised  of  Best  Practices  and  Lessons 
Learned. 

The  final  portion  of  the  DPAP  database  of  peer-review  results  to  be  examined 
in  relation  to  Recommendations  is  by  Contract  Type.  Figure  14  lists  the  number  of 
occurrences  of  Recommendations  within  each  listed  Contract  Type. 
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Occurrences  of  Recommendation  Type  Feedback  by 
Contract  Type 

Weapon  System  Technology  Demonstration 
Weapon  System  Production  Lot  Buy  (Sole  Source) 

Weapon  System  Development  Sole  Source 
Weapon  System  Development 
Sole  Source  Procurement  of  Services 
Non  Competitive  Weapon  System  Development 
Non  Competitive  Weapon  System 
Non  Competitive  Supplies  Contract 
Non  Competitive  Services  Contract 
Non  Competitive  Multiple  Award  Services  Contract 
Multiple  Award  Contract  for  R&D 
Multiple  Award  Construction  Contract 
Logistical  Services  (Competitive) 

Design/Bulld  Construction  Contract 
Competitive  Weapons  System  Development 
Competitive  Supplies  Contract 
Competitive  Services  Contract 
Competitive  Procurement  of  Supplies 
Competitive  Procurement  of  Services 
Competitive  Multiple  Award  IDIQ  Supplies  Contract 
Competitive  Multiple  Award  IDIQ 
Competitive  Multiple  Award  Services  Contract 
Competitive  Multiple  Award  IDIQ  Services 
Competitive  Multiple  Award  IDIQ  R&D  Contract 
Competitive  Multiple  Award  (Combination  Multiple  Award...  7 

Competitive  IQC  |  1 
Competitive  IDIQ  Services  Contract  ■  2 
Competitive  IDIQfor  IT  Products  4 
Competitive  IDIQ  I  1 
Competitive  Design/Bulld  FPIF  ■  2 
Commodity  (Competitive)  |B|  3 

0  10  20  30  40  50 

■  Occurrences  of  Recommendation 

Figure  14.  Frequency  Distribution  of  Recommendations  by  Contract  Type 

(Adapted  from  DPAP,  2013) 
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Similarly  to  the  way  the  data  for  recommendations  within  each  category  was 
dissected,  Table  2  provides  a  breakdown  of  percentage  of  each  contract  type  that 
returned  a  recommendation. 

Table  2.  Percentage  of  Recommendations  by  Contract  Type 

(Adapted  from  DPAP,  2013) 


Contract  Type 

Total 

Occurrences  of 
Recommendation 

Percentage  of 
Recommendation 

Commodity  (Competitive) 

4 

3 

75% 

Competitive  Design/Build  FPIF 

2 

2 

100% 

Competitive  IDIQ 

1 

1 

100% 

Competitive  IDIQ  for  IT  Products 

4 

4 

100% 

Competitive  IDIQ  Services  Contract 

6 

2 

33.33% 

Competitive  IQC 

1 

1 

100% 

Competitive  Multiple  Award  (Combination 

7 

7 

100% 

Multiple  Award  FFP  and  Cost  type  line  items) 

Competitive  Multiple  Award  IDIQ  R&D 

1 

0 

0% 

Contract 

Competitive  Multiple  Award  IDIQ  Services 

22 

3 

13.64% 

Competitive  Multiple  Award  Services  Contract 

59 

41 

69.49% 

Competitive  Multiple  Award  IDIQ 

5 

2 

40% 

Competitive  Multiple  Award  IDIQ  Supplies 

2 

1 

50% 

Contract 

Competitive  Procurement  of  Services 

20 

9 

45% 

Competitive  Procurement  of  Supplies 

14 

9 

64.29% 

Competitive  Services  Contract 

39 

16 

41.03% 

Competitive  Supplies  Contract 

1 

1 

100% 

Competitive  Weapons  System  Development 

16 

14 

87.5% 

Design/Build  Construction  Contract 

3 

3 

100% 

Logistical  Services  (Competitive) 

10 

7 

70% 

Multiple  Award  Construction  Contract 

2 

2 

100% 

Multiple  Award  Contract  for  R&D 

6 

5 

83.33% 

Non-Competitive  Multiple  Award  Services 

11 

11 

100% 

Contract 

Non-Competitive  Services  Contract 

4 

4 

100% 

Non-Competitive  Supplies  Contract 

10 

8 

80% 

Non-Competitive  Weapon  System 

8 

8 

100% 

Non-Competitive  Weapon  System 

6 

5 

83.33% 

Development 

Sole-Source  Procurement  of  Services 

1 

1 

100% 

Weapon  System  Development 

5 

4 

80% 

Weapon  System  Development  Sole  Source 

1 

0 

0% 

Weapon  System  Production  Lot  Buy  (Sole 

3 

3 

100% 

Source) 

Weapon  System  Technology  Demonstration 

14 

13 

92.86% 

Total 

288 

190 

65.97% 

The  variety  of  contract  types  listed  makes  it  difficult  to  pinpoint  precise  issues 
within  the  Contract  Type  data  element.  If  we  aggregate  the  Service  type  contracts, 
then  the  rate  of  Recommendations  is  49.47%.  This  does  not  represent  an 
overwhelming  rate  given  the  concentration  of  Recommendation  Type  Feedback  that 
some  data  elements  in  the  DPAP  database  of  peer-review  results  have  experienced 
Scanning  Table  2  would  seem  to  indicate  that  the  Contract  Type  data  element  is 
garnering  more  than  its  fair  share  of  Recommendations,  indeed,  rates  upwards  of 
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70%  seem  prevalent.  Two  Contract  Type  data  elements  are  drastically  pulling  the 
average  down.  Competitive  Multiple  Award  IDIQ  Services  with  three 
Recommendations  in  22  occurrences,  and  Competitive  Procurement  of  Services 
with  nine  Recommendations  in  20  occurrences.  With  the  wide  range  of  Contract 
Types  employed  in  this  data  element,  it  will  take  more  entries  in  the  DPAP  database 
of  peer-review  results  to  draw  meaningful  conclusions  about  where  issues  with 
Contract  Type  exist.  The  next  section  will  address  comparisons  between  the 
Contracting  Core  Competencies  and  the  DPAP  database  of  peer-review  results. 

C.  CONTRACTING  CORE  COMPETENCIES  COMPARISONS 

The  DoD  CCM  was  developed  in  2007  after  an  exhaustive  survey  was 
conducted  across  multiple  DoD  contracting  organizations.  The  survey  asked  both 
employees  and  supervisors  to  assess  critical  areas  within  the  contracting  process 
they  worked  on  and  also  asked  them  to  rate  themselves  from  a  proficiency 
standpoint.  Subject  matter  experts  from  the  contracting  career  field  took  a  list  of  27 
business  and  professional  competencies  and  were  asked  to  rank  them  as  to  how 
effective  they  were  in  relation  to  job  performance.  Ten  competencies  were  identified 
as  the  most  important  for  work  performance.  From  that,  the  CCM  was  developed, 
consisting  of  12  units  of  competence,  28  technical  competencies,  and  10 
professional  competencies  (DPAP,  2007).  Table  3  represents  the  CCM. 
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Table  3.  DoD  Contracting  Competency  Model 

(DPAP,  2007) 


DoD  Contracting  Competency  Model 
established  March  2007 


Pre-Award  aivd  Award 

Detenrinatpon  of  How  Best  To  Satisfy  Requirements 

Considers  Soao-econonic  Requirenaefig 

Promote  COTpeSon 

Source  Sefecwn  Ptamting 

Sofcdanon  of  Offers 

ResponsMy  Detemnnatiort 

Bid  Evaluation  (Seated  Mtftng) 

Proposal'  Evaluation  (Cofitacing  by  Megofi^on) 

Source  Setedion 

ContraaAwafd 

Process  Protests 

Develop  artcl/or  Negcflisoe  Positions 

Justifcaoon  of  Ottier  than  FiJ  and  Open  Comipeijison 

Terms  and  Conditions 

Preparation  and  Negotiation 

Advanced  Coscarid/or  Pto  Analyse 

Advanced  Cost  and/or  Price  Analysis 

Cmraa  Adnweiraftori 

Inoaaon  of  worK 

Contraa  Performance  ManagemefS 
issue  Changes  and  Medications 

Approve  Faymem  Request 

Ctose-out  Contracts 

Smal  Biisiness/SockyEocHnoffiic  Programs 

Addressing  Smal  Busmess  Concerns 

Negotiate  Forward  Prtang  Rale  Agreemeints 
and  Admreter  Cosit  Accovniing  Standards 

Negotiate  Forward  Pricing  Rate  Agreements 
and  Adnmister  Co^  Accounting  Standards 

Coffwact  Terrmiaiion 

Contract  TentinaftOn 

Proair^nent  Pofcy 

Procurement 

Otier  Competencies 

E-Busmess  and  Automated  Tools 

AcMy  Program  Coordinator  for  Purchase  Card 

Construdon/AtcMea  and  Engmeering 

Contracting  in  a  ConSngefrt  and/or  Combat  Environment 

Contracting  in  a  Contingent  and/or  Combat  Environment 

Professional  Coir^petency 

Problem  Sot^mg 

Customer  Service 

Oral  Communication 

Written  Cornmunfoatiof! 

Interpersonal  SIJs 

Decisiveness 

Technical  Gredib% 

Ftex<)«y 

Resience 

Accounmbiiry 
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The  final  report  detailing  the  survey  results  and  providing  the  model  was 
entitled  Department  of  Defense  Contracting  Workforce  Competency  Assessment 
Final  Report  (DPAP,  2007).  Included  in  the  report  were  high  risk  areas  within  the 
contracting  workforce  that  had  been  identified  through  the  survey  process.  These 
high  risk  areas  were  described  as  “Competency  Gaps.”  The  following  is  a  list  of  the 
areas  established  as  Competency  Gaps: 

•  fundamental  contracting  skills  across  entry  and  journey  levels  of  the 
contracting  workforce  and  currency,  breadth,  and  depth  of  knowledge 
across  journey  and  senior  levels; 

•  the  source  selection  process 

•  cost  and  price  analysis 

•  contract  performance  management 

•  integrated  acquisition  skills  (DPAP,  2007). 

From  these  competency  gaps,  a  few  areas  from  the  DPAP  database  of  peer- 
review  results  can  be  cross-referenced.  Within  the  DoD  CCM,  the  source  selection 
process  is  broken  down  into  the  following  subparts:  source  selection  planning, 
source  selection,  proposal  evaluation,  and  contract  award.  Referencing  the  previous 
section  on  the  Recommendation  Feedback  Type  breakdown  by  phase,  the  greatest 
frequency  of  Recommendations  by  phase  was  in  Phase  1.  Phase  1  would  correlate 
with  source-selection  planning  within  the  CCM,  but  since  the  Department  of  Defense 
Contracting  Workforce  Competency  Assessment  Final  Report  {DPAP,  2007)  did  not 
break  out  any  of  the  sub-areas  within  the  source  selection  process  as  being  any 
more  at  risk  than  any  other  area,  it  is  difficult  to  ascertain  whether  there  is  a  specific 
relationship  between  the  report  results  and  the  observations  from  the  DPAP 
database  of  peer-review  results.  The  obvious  spike  in  Recommendations  in  Phase  1 
can  be  generalized  as  an  issue  with  the  overall  source-selection  process,  which 
helps  to  corroborate  the  survey  results  on  which  the  CCM  was  based. 

Two  areas  from  the  DPAP  database  of  peer-review  results  that  can  be  more 
directly  related  to  the  CCM  are  the  Pricing  and  Incentive  &  Award  Fee  categories. 
Recall  that  of  the  three  Types  of  Feedback,  Recommendations  comprised  89.13%  of 
the  feedback  for  the  Pricing  category  and  86.96%  of  the  feedback  for  the  Incentive  & 
Award  Fee  category.  These  two  categories  garnered  the  highest  densities  of 
Recommendations  within  the  DPAP  database  of  peer-review  results,  providing  an 
indication  that  these  areas  are  presenting  difficulties  to  the  contracting  workforce. 
This  also  shows  a  relationship  to  the  Competency  Gaps  of  cost  and  price  analysis. 
Unfortunately,  without  having  dates  associated  with  the  DPAP  database  of  peer- 
review  entries,  it  is  difficult  to  ascertain  whether  the  training  initiatives  that  were 
driven  by  the  Department  of  Defense  Contracting  Workforce  Competency 
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Assessment  Final  Report  (DPAP,  2007)  are  having  a  noticeable  impact  on  the 
number  of  Recommendations  that  these  two  categories  are  attracting  from  the  peer- 
review  teams. 

The  final  area  identified  as  a  Competency  Gap  that  has  a  cross-reference 
with  the  DPAP  database  of  peer-review  results  is  the  number  of  Recommendations 
within  the  Post-Award  1  Review  Phase,  albeit  tentatively  so.  Contract  performance 
management  was  identified  as  Competency  Gap,  which  would  align  with  the  Post- 
Award  1  Review  Phase  of  the  DPAP  process.  The  number  of  Recommendations 
showed  an  uptick  between  Phase  4,  with  no  recommendations,  to  11 
Recommendations  in  the  Post-Award  1  Phase.  The  link  remains  tenuous,  however, 
and  more  contracts  will  need  to  be  reviewed  in  the  Post-Award  1  Phase  to 
strengthen  the  tie  between  the  peer-review  results  and  the  Competency  Gap.  The 
following  section  addresses  themes  within  the  narrative  feedback  data  element  of 
the  DPAP  database  of  peer-review  results. 

D.  NARRATIVE  FEEDBACK  THEMES 

The  narrative  feedback  data  element  of  the  DPAP  database  of  peer-review 
results  provides  an  opportunity  for  the  peer-review  team  to  enter  its  comments  on 
the  occurrence  documented.  Although  difficult  to  summarize  or  quantify  the 
narratives,  some  themes  did  occur  within  the  comments. 

The  most  startling,  if  not  alarming,  issue  is  the  episodes  that  reveal  direct 
violations  of  the  FAR  or  DFARS.  Concerns  existed  about  improperly  proposed 
award  fees,  award  fees  that  were  not  necessary,  and  allowing  offerors  to  propose 
their  own  fee  pool  percentage.  In  another  instance,  a  contracting  officer  attempted  to 
provide  a  performance  incentive  for  a  project  in  progress  review  that  was  already 
mandated  by  a  DFARS  clause.  Proposals  with  vague  requirements  were  also  a 
recurring  theme. 

Questions  overtraining  adequacies  persisted  as  well.  Some  comments 
revolved  around  concerns  over  the  use  of  contracting  officer  representatives  and 
whether  they  had  received  proper  training  to  execute  their  intended  function.  One 
training  comment  stood  out  above  the  others:  “Minimal  source  selection  experience 
within  the  government  evaluation  team  has  resulted  in  heavy  reliance  on  contractor 
expertise.  Recommend  obtaining  assistance  from  the  DAD  in  developing  the  source 
selection  framework”  (DPAP,  2013). 

The  narrative  comments  read  like  a  summary  of  GAO  and  Inspector  General 
(IG)  reports  that  have  detailed  the  deficiencies  with  contracting  over  the  last  several 
years.  The  disturbing  aspect  is  that  these  entries  represent  contracts  valued  at  $1 
billion  and  above.  Presumably,  with  such  a  high  dollar  value,  DoD  agencies  and 
departments  would  have  experienced  contracting  officers  and  dedicated  contracting 
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teams  developing  these  contracts.  Yet  mistakes  that  would  be  associated  with  an 
inexperienced  contracting  team  are  appearing  in  the  feedback.  The  next  section 
contains  recommendations  for  improvements  to  the  DPAP  database  of  peer-review 
results. 

E.  RECOMMENDATIONS  FOR  IMPROVEMENTS  TO  THE  DPAP  DATABASE 
OF  PEER-REVIEW  RESULTS 

1.  Introduction 

This  section  addresses  recommendations  for  improvements  to  the  DPAP 
database  of  peer-review  results  based  on  the  data  analysis  conducted.  The  following 
recommendations  would  improve  the  functionality  and  usefulness  of  the  DPAP 
database  of  peer-review  results.  An  additional  category  should  be  added  to  the 
DPAP  database  of  peer-review  results  to  provide  what  basic  contract  type  is  being 
utilized  in  the  entry.  The  existing  Contract  Type  data  element  should  be  repurposed 
as  a  Requirement  Type  data  element.  The  Type  of  Feedback  data  element  needs  to 
add  another  category  to  address  regulatory  requirements.  The  Source  Selection 
Category  should  be  further  subdivided  to  address  the  phases  that  make  up  the 
source  selection  process.  Lastly,  labeling  conventions  across  DPAP  produced 
products  should  be  consistent. 

2.  Addition  of  a  Data  Eiement 

The  first  recommendation  is  to  add  another  data  element  to  the  DPAP 
database  of  peer-review  results.  Currently,  there  is  a  Contract  Type  data  element, 
but  it  does  not  simply  list  the  commonly  used  types  of  cost  and  fixed-type  contracts. 
The  current  Contract  Type  field  should  be  relabeled  as  Contract  Requirement  Type 
to  retain  descriptive  value  and  context  for  the  entry  being  examined.  The  Contract 
Type  field  should  be  repurposed,  containing  one  of  the  basic  contract  types.  As  the 
DPAP  database  of  peer-review  results  stands,  the  state  of  the  Contract  Type  field 
negatively  impacts  data  analysis.  This  is  due  to  the  difficulty,  if  not  down-right 
impossibility,  of  identifying  which  basic  contract  type  is  being  employed  in  regards  to 
the  data  entry.  If  basic  contract  types  were  listed,  then  basic  analytics  could  be 
conducted  to  identify  which,  if  any,  contract  types  are  garnering  more  negative 
feedback  than  others. 

3.  Regulatory  Requirements 

The  second  recommendation  addresses  the  Recommendation  feedback  type. 
The  Recommendations  make  up  the  majority  of  the  DPAP  database  of  peer-review 
results  entries  and  address  a  myriad  of  issues,  some  small,  some  large.  The  issue 
that  needs  to  be  addressed  is  Recommendations  that  are  regulatory  in  nature.  For 
example,  consider  this  entry  “An  Award  Fee  plan  needs  to  conform  with  the  FAR 
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Rule  16.401(e)”  (DPAP,  2013).  This  Recommendation  targets  a  violation  of  a 
regulation,  in  this  case  FAR  16.401(e);  thus,  this  particular  Recommendation  carries 
far  more  weight  than  might  normally  be  associated  with  a  recommendation.  To  call 
attention  to  regulatory  violations,  the  DPAP  database  of  peer-review  results  needs  to 
contain  a  “regulatory”  Feedback  Type  that  can  be  used  to  pinpoint  such  violations.  It 
would  provide  another  method  for  policy-makers  to  determine  over  time  which  areas 
of  the  FAR  are  habitually  misunderstood  or  underutilized  so  training  can  be  tailored 
to  address  those  shortcomings. 

4.  Subdivide  the  Source  Selection  Category 

The  Source  Selection  Category  would  benefit  from  additional  refinement, 
adding  to  the  overall  usefulness  of  the  DPAP  database  of  peer-review  results.  The 
Source  Selection  Category  contained  115  entries  out  of  the  288  total  entries  in  the 
database.  By  comparison.  Pricing  came  in  a  distant  second  at  46  entries.  Source 
Selection  as  it  is  being  used  currently  within  the  DPAP  database  of  peer-review 
results  covers  four  distinct  phases  of  the  contracting  process.  Those  phases  are 
procurement  planning,  solicitation  planning,  solicitation,  and  source  selection.  The 
Source  Selection  category  should  be  further  subdivided  into  categories  that  mirror 
the  aforementioned  phases,  again  for  the  purposes  of  providing  an  accurate 
assessment  of  where  problems  are  occurring  within  the  contracting  process  and 
enabling  policy-makers  to  address  deficiencies. 

5.  Ensure  Labeling  Conventions  Are  Consistent 

The  final  recommendation  addresses  labeling  conventions.  Phases,  titles, 
contract  types,  and  so  forth  within  the  DPAP  database  of  peer-review  results  do  not 
consistently  match  up  well  with  other  contracting-related  products,  some  of  which 
were  created  by  the  DPAP  organization  itself.  For  example,  it  is  difficult  to  cross 
reference  any  area  of  the  DPAP  peer-review  database  with  the  CCM.  The  CCM  was 
developed  by  DPAP  shortly  before  the  peer-review  requirement  was  established  in 
2008.  Assuming  the  DoD  is  serious  about  improving  the  performance  of  its 
contracting  workforce,  then  should  not  labels  on  the  DPAP  database  of  peer-review 
results  allow  a  user  to  cross-reference  competencies  that  have  been  identified  as 
critical  for  accomplishing  contracting  functions?  The  subsequent  section 
encompasses  the  summary  of  the  data  analysis  chapter. 

F.  SUMMARY 

This  chapter  discusses  the  results  of  the  analysis  of  the  DPAP  peer  reviews. 
Trends  within  the  DPAP  database  of  peer-review  results  were  examined. 
Frequencies  of  occurrences  by  Category,  Type  of  Contract,  Review  Phase  and  Type 
of  Feedback  were  inspected.  The  Recommendation  Type  of  Feedback  received 
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additional  attention.  The  data  showed  the  majority  of  feedback  within  the  DPAP 
database  of  peer-review  results  occurred  in  the  Recommendation  Feedback  Type, 
and  in  the  initial  Review  Phases.  Further  examination  reveals  that  the  majority  of  the 
occurrences  of  the  Pricing,  Incentive  and  Award  Fee,  &  Requirements/PWS/SOW 
Categories  consist  of  Recommendation  Feedback  Type.  Applicability  of  relationships 
within  the  data  to  previously  identified  Competency  Gaps  within  the  DoD  contracting 
workforce  was  examined.  Themes  were  identified  within  the  narrative  feedback 
comments  that  showed  issues  requiring  corrective  action  within  high  dollar  DoD 
contracts.  The  next  chapter  summarizes  and  concludes  the  research  and  provides 
areas  for  further  research. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-43- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


V.  SUMMARY,  CONCLUSIONS,  AND  AREAS  FOR  FURTHER 

RESEARCH 


A.  SUMMARY 

Since  the  early  1990s,  the  number  of  personnel  within  the  acquisition 
workforce  has  steadily  declined.  Since  the  onset  of  the  overseas  contingency 
operations  in  2002,  DoD  contracting  actions  have  risen  astronomically.  Additionally, 
since  the  reliance  on  contracts  has  increased,  DoD  shows  no  signs  of  decreasing 
that  reliance  for  accomplishing  its  mission.  With  the  reduction  in  the  workforce  and 
the  rapid  increase  in  the  number  of  contract  actions  and  the  amount  of  dollars 
expended,  issues  within  the  contracting  workforce  have  also  risen  dramatically, 
mostly  as  a  result  of  a  workforce  that  is  either  too  inexperienced  or  spread  too  thin 
for  its  assigned  task.  DPAP  implemented  the  independent  management  review  (or 
peer  review)  requirement  in  2008  as  a  means  of  attempting  to  compensate  for  the 
lack  of  experienced  contracting  personnel  within  its  ranks.  The  next  section 
examines  the  conclusions  of  this  thesis. 

B.  CONCLUSIONS 

Research  Question  1:  Are  there  trends  within  the  peer-review  results  of 
DoD-level  peer-reviewed  contracts? 

Trends  were  identified  in  the  DPAP  database  of  peer-review  results.  Not 
surprisingly,  most  of  the  entries  consisted  of  Recommendation  Type  Feedback,  at 
65%  of  the  entries;  however.  Best  Practice  Feedback  Type  was  represented  at  24% 
and  Lessons  Learned  at  11%.  This  demonstrates  the  DPAP  database  of  peer-review 
results  potential  as  a  tool  to  spread  effective  contracting  practices  throughout  DoD. 
The  analysis  also  showed  that  the  entries  were  largely  Source  Selection  focused, 

110  out  of  288  entries  pertained  to  Source  Selection.  The  next  most  prevalent 
occurrence  within  Category  Type  was  Pricing  with  46  entries.  Potential  issues  were 
also  indicated  with  the  following  Categories:  Terms  &  Conditions, 
Requirements/PWS/SOW,  and  Incentive  &  Award  Fee.  Those  categories  amassed 
70%  or  greater  Recommendation  Type  Feedback. 

Research  Question  2:  Are  any  trends  identified  related  to  the 
competency  gaps  identified  in  the  Department  of  Defense  Contracting 
Workforce  Competency  Assessment  Final  Report? 

In  regards  to  trends  between  the  DPAP  database  of  peer-review  results  and 
competency  gaps  from  the  Department  of  Defense  Contracting  Workforce 
Competency  Assessment  Final  Report  (DPAP,  2007),  the  strongest  link  was 
between  issues  with  Cost  and  Pricing  Analysis  identified  in  the  Final  Report  and  the 
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DPAP  database  of  peer-review  results  Pricing  and  Incentive  &  Award  Fee 
Categories.  Pricing  consisted  of  89%  Recommendation  Feedback  Type,  and 
Incentive  &  Award  Fee  consisted  of  87%. 

This  research  identified  trends  within  the  results  provided  by  the  peer-review 
teams  and  recorded  within  the  DPAP  database  of  peer-review  results.  The  results  of 
the  study  can  be  used  for  further  research  into  areas  such  as  identifying  recurring 
problem  areas,  identifying  training  gaps,  and  answering  other  “why”  type  questions 
associated  with  the  results  of  this  study.  This  study  was  by  no  means  intended  to  be 
exhaustive  in  nature,  but  rather  to  investigate  an  area  that  had  previously  received 
little,  if  any,  attention. 

The  DPAP  database  of  peer-review  results  is  a  good  tool  for  the  DoD 
contracting  workforce;  it  does  require  some  refinement  to  reach  its  full  potential,  as 
well  as  the  implementation  of  a  dedicated  feedback  mechanism  to  disseminate  the 
peer-review  results  back  to  the  contracting  workforce.  An  important  consideration 
that  comes  into  focus  is  the  fact  that  the  trends  show  the  same  issues  facing  lower 
dollar  threshold  contracting  actions  are  impacting  high  dollar  contracting  actions  as 
well.  However,  since  the  DPAP  database  of  peer-review  results  threshold  level  is 
above  $1  billion,  the  effects  of  mistakes  in  contracting  are  magnified. 

C.  AREAS  FOR  FURTHER  RESEARCH 

As  entries  in  the  DPAP  database  of  peer-review  results  accumulate  over  time, 
the  statistical  population  for  analysis  will  increase.  The  current  number  of  entries 
within  the  DPAP  database  of  peer-review  results  allows  for  trend  and  frequency 
analysis  within  the  database  itself.  However,  once  the  various  fields  are  broken 
down  into  their  constituent  parts,  more  entries  would  allow  for  more  meaningful 
comparisons. 

Specifically,  there  are  two  areas  to  consider.  First,  as  Best  Practice  feedback 
types  continue  to  accumulate,  is  there  a  relationship  with  the  Contract  Management 
Maturity  Model  (CMMM)?  CMMM  assessments  contain  assessments  of  process 
strength,  process  results,  and  management  support.  Do  the  narrative  comments  in 
the  DPAP  database  of  peer-review  results  best  practice  entries  support  any  of 
those?  Additionally,  if  service-level  peer-review  results  could  be  obtained,  a  more 
direct  comparison  could  be  made  with  respective  CMMM  assessments  from  that 
service.  Second,  since  the  DPAP  database  of  peer-review  results  is  a  relatively  new 
construct,  as  more  contracts  enter  the  Post  Award  1  review  phase,  will  we  continue 
to  see  an  uptick  in  recommendations  (see  Figure  1 1)7  Could  this  indicate  a 
continuation  of  poor  contractor  surveillance  practices  previously  noted  by  the  GAO? 

The  Source  Selection  category  received  the  most  attention  overall  by  the 
peer-review  teams  with  115  out  of  288  entries.  The  source  selection  process 
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receives  a  good  deal  of  emphasis  in  training  and  in  practice  by  the  contracting 
community,  while  the  contract  administration  phase  is  sometimes  considered  to  be 
neglected.  An  interview-based  study  with  members  of  peer-review  teams  could 
potentially  identify  whether  the  Source  Selection  category  receives  so  much 
attention  in  the  DPAP  database  of  peer-review  results  because  it  is  simply  better 
understood  than  the  other  categories. 

The  peer-review  process  will  continue  to  evolve  at  the  DoD  level  and  the 
service  component  level.  As  the  process  evolves,  the  lessons  provided  by  the  peer- 
review  process  should  be  disseminated  back  out  to  the  contracting  workforce.  As 
this  occurs,  will  sustained  protest  rates  decrease? 

The  service  components  are  mandated  by  the  peer-review  policy  to  execute 
peer  reviews  at  the  service  level  for  lower  dollar  threshold  contract  actions.  Are  the 
services  complying  with  this  requirement?  Are  they  compiling  a  database  similar  to 
the  DPAP  database  of  peer-review  results?  Can  the  peer-review  results  be  obtained 
for  further  analysis?  The  Army  FAR  supplement,  the  AFARS,  provides  more  detail 
on  the  peer-review  requirement.  The  AFARS  assigns  responsibility  for  peer  reviews 
of  solicitations  and  contracts  valued  at  greater  than  $50  million  to  a  Solicitation 
Review  Board  (SRB)  and  Contract  Review  Board  (CRB).  The  Principal  Assistant 
Responsible  for  Contracting  (PARC)  has  responsibility  for  establishing  peer-review 
procedures  for  contracts  valued  at  less  than  $50  million. 

The  AFARS  51 01.1 70(1  )(c)  also  mandates  that 

The  SRB/CRB  will  be  an  independent,  multi-functional  team  comprised 
of  senior  level  experts,  which  will  at  a  minimum  include  representatives 
from  the  acquisition  center,  small  business  office,  office  of  counsel, 
requirements  community,  and  in  the  case  of  non-competitive  actions, 
the  competition  advocate.  (AFARS,  2014) 

From  that  definition,  it  would  seem  evident  that  the  Army  is  attempting  to 
garner  insights  from  all  angles  of  the  acquisition  process  into  the  peer-review 
process.  The  AFARS  also  stipulates  that  the  same  members  will  take  part  in  both 
the  SRB  and  CRB;  although  a  good  idea,  procurement  lead-times  might  often 
prevent  this  from  happening,  given  typical  military  permanent  change  of  station 
(PCS)  cycles.  There  is  also  strict  language  on  who  will  chair  the  boards,  and  there  is 
no  ability  to  delegate  the  authority.  The  PARC  is  required  to  chair  SRB/CRB  for 
actions  valued  between  $50  million  and  $250  million,  while  the  Head  of  the 
Contracting  Activity  (HCA)  has  responsibility  for  the  $250  million  to  $1  billion  range. 
Interestingly,  per  AFARS  5101 .170(1  )(e),  PARCs  and  the  HCA  may  waive  the 
requirement  for  a  formal  review  board.  The  waiver  is  required  to  be  in  writing  and 
included  in  the  contract  file.  Data  on  the  number  of  peer  reviews  conducted,  along 
with  any  issues  or  trends,  best  practices,  or  relevant  feedback  on  the  process  itself, 
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is  required  to  be  submitted  quarterly  (along  with  information  on  waivers  of  the  review 
process)  to  the  Procurement  Policy  and  Support  Directorate  (AFARS,  2014). 

AFARS  5101.170(2)  continues  with  further  direction  on  the  handling  of  post¬ 
award  peer  reviews.  Post-award  peer  reviews  for  contracts  valued  at  between  $500 
million  and  $1  billion  will  be  conducted  for  services  contracts  that  were  approved  by 
a  Headquarters,  Department  of  the  Army  (HQDA)-level  Army  Services  Strategy 
Panel  (ASSP).  The  post-award  review  minimum  review  requirements  are  as  follows: 

•  Contract  performance  in  terms  of  cost,  schedule,  and  requirements. 

•  Use  of  contracting  mechanisms,  including  the  use  of  competition,  the 
contract  structure  and  type  the  definition  of  contract  requirements,  cost 
or  pricing  methods,  the  award  and  negotiation  of  task  orders,  and 
management  and  oversight  mechanisms. 

•  Contractor’s  use,  management,  and  oversight  of  subcontractors. 

•  Staffing  of  contract  management  and  oversight  functions 

•  Extent  of  any  pass-through  and  excessive  pass-through  charges  by 
the  contractor  (as  defined  in  Section  852  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2007,  Public  Law  109-364). 

ASSP  panel  members  are  to  address  whether  or  not  the  services  review 
structure  is  effective,  evaluate  the  current  acquisition  as  to  adequacy  of  competition, 
provide  an  assessment  of  contract  performance,  assess  the  conduct  of  the 
government  surveillance  plan,  and  pass  on  any  lessons  learned  or  best  practices  for 
use  on  other  ongoing  acquisitions  (AFARS,  2014).  Additionally,  the  Army 
Contracting  Command  Desk  Book  (May  2012  edition),  the  handbook  for  Army 
contracting  officers,  provides  the  requirements  for  peer  reviews  listed  from  the 
AFARS.  The  only  additional  guidance  it  provides  are  two  toolkits  for  pre-award  and 
post-award  peer  reviews.  The  toolkits  list  documents  required  by  the  OSD-level 
peer-review  team  and  elements  required  to  be  confirmed  by  the  OSD  peer-review 
team  (ACC,  2012). 

Finally,  interviews  could  be  conducted  with  DPAP  personnel  to  obtain  time- 
phased  data  about  entries  in  the  DPAP  database  of  peer-review  results.  The  data 
could  then  be  analyzed  to  determine  if  the  competency  gaps  identified  in  the 
Department  of  Defense  Contracting  Workforce  Competency  Assessment  Final 
Report  (DPAP,  2007)  are  closing. 
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